feat: add armhf (32-bit ARM) architecture support [ISD-5856] by florentianayuwono · Pull Request #233 · canonical/github-runner-image-builder-operator

florentianayuwono · 2026-07-02T07:16:00Z

Overview

Adds armhf (32-bit ARM / linux-arm) as a buildable image architecture so the charm can
produce GitHub Actions runner images that run the 32-bit runner agent on arm64 (aarch64)
hardware — no emulation.

The key design point: an armhf image is an arm64 boot image (64-bit kernel) that carries a
32-bit linux-arm runner payload plus an armhf multiarch userland. The 32-bit runner executes
via the host's native AArch32 support (AArch32 at EL0). A native armhf image (32-bit kernel) does
not boot on the aarch64 virt machine, which is why the boot/glance architecture is aarch64
while the runner payload is 32-bit arm.

The runner binary is the linux-arm build published by the
canonical/github-actions-runner fork
(FORK_RUNNER_BINARY_REPO): actions-runner-linux-arm-<version>.tar.gz (arm32 actions wired onto
Node.js 22, from fork PRs #155/#169/#171).

How an armhf image is built

Boot arch ≠ payload arch. Arch.ARM.to_openstack() → aarch64 and the base cloud image
download arch (_get_supported_runner_arch) → arm64, so the base image and the resulting
snapshot schedule and boot on arm64 hosts. RUNNER_ARCH renders to arm, so cloud-init
downloads actions-runner-linux-arm-<version>.tar.gz.
armhf multiarch userland (enable_armhf_multiarch in cloud-init.sh.j2, armhf-only):
- dpkg --add-architecture armhf.
- Pin the stock ubuntu.sources to the native arch (Architectures: <native>), because the
  arm64 mirrors only serve the native arch.
- Add an armhf-ports.sources pointing at ports.ubuntu.com (armhf is a "ports" architecture),
  so the 32-bit runtime dependencies resolve.
- Write /etc/cloud/cloud.cfg.d/99-armhf-preserve-apt-sources.cfg (preserve_sources_list: true)
  so the pinned/ports sources survive on the runner VM (see "Why preserve_sources_list").
armhf runtime + build deps (ARM_ADDITIONAL_APT_PACKAGES, installed only for Arch.ARM):
libc6:armhf (provides ld-linux-armhf.so.3), libatomic1:armhf, rustup, docker-buildx,
plus the release-specific armhf libicu (libicu74:armhf on noble, libicu78:armhf on
resolute — ARM_LIBICU_APT_PACKAGE_BY_BASE). The distro cargo/rustc are dropped
(ARM_EXCLUDED_DEFAULT_APT_PACKAGES) because rustup provides them and conflicts otherwise.
libicu + libatomic are the runner's bundled .NET runtime dependencies.
Rust toolchain: cloud-init runs sudo -u ubuntu rustup default stable (armhf-only; a no-op
elsewhere since rustup is only installed on armhf images).

Why `preserve_sources_list: true`

Before snapshotting, the builder runs cloud-init clean --logs --machine-id --seed --configs all.
On the runner VM's first boot cloud-init re-runs and regenerates ubuntu.sources, dropping the
Architectures: pin — while the dpkg --add-architecture armhf state persists. Apt would then
request binary-armhf from the arm64-only primary/security mirrors and 404 on apt-get update.
The drop-in lives in /etc/cloud/cloud.cfg.d/ (which cloud-init clean does not touch) and
tells cloud-init not to manage apt sources, so the baked multiarch config survives.

armhf is restricted to noble+ base images

rustup, docker-buildx and the pinned libicu sonames are only available from noble (24.04)
onwards. _generate_cloud_init_script raises UnsupportedArchitectureError up front when
arch == ARM and the base is older than noble (ARM_SUPPORTED_BASE_IMAGES = (NOBLE, RESOLUTE)),
instead of failing midway through the build.

Changes

Charm layer

src/state.py: add ARM to the Arch enum and ARCHITECTURES_ARM = {"armhf", "arm", "armv7l"};
map those config values to Arch.ARM in Arch.from_charm.
charmcraft.yaml: document the armhf/arm architecture option (experimental; notes it builds
on arm64 hardware with a 32-bit userspace and uses the fork's runner binary).

App layer

config.py: add Arch.ARM; to_openstack() → aarch64; ARM_ADDITIONAL_APT_PACKAGES,
ARM_EXCLUDED_DEFAULT_APT_PACKAGES, ARM_LIBICU_APT_PACKAGE_BY_BASE.
cloud_image.py: Arch.ARM base download arch → arm64; add armhf to
SupportedBaseImageArch.
openstack_builder.py: assemble the armhf apt package set (drop cargo/rustc, add armhf multiarch
libs + release-matched libicu); fail fast with UnsupportedArchitectureError on pre-noble bases;
ARM_SUPPORTED_BASE_IMAGES.
templates/cloud-init.sh.j2: enable_armhf_multiarch() (dpkg arch, source pinning,
ports.ubuntu.com source, preserve_sources_list drop-in) run before package install for
arch == arm; rustup default stable for the ubuntu user.
store.py: minor refactor of the image-properties dict (no behaviour change).

Tests

Unit (TDD) — app suite: 133 passed, black/isort/flake8 clean.

test_config.py — Arch.ARM → "aarch64".
test_cloud_image.py — Arch.ARM base download arch → arm64.
test_state.py — armhf/arm/armv7l → Arch.ARM.
test_openstack_builder.py::test__generate_cloud_init_script — armhf on a supported base renders
the multiarch setup, the correct apt set (cargo/rustc removed; armhf libs + matched libicu added),
and the rustup step.
test_openstack_builder.py — armhf on focal/jammy raises UnsupportedArchitectureError.
test_integration_commands.py — commands_for_arch(Arch.ARM) includes the armhf assertions.

Integration — a new armhf matrix leg (noble + resolute) in
.github/workflows/integration_test_app.yaml, sharing the arm64 tenant creds. It builds a real
snapshot, boots a runner VM from it, and runs the base runner command suite plus armhf-specific
assertions (ARM_RUNNER_COMMANDS): rustc/cargo/docker buildx present, and crucially
file .../Runner.Listener is an ELF 32-bit ... ARM binary (interpreter
/lib/ld-linux-armhf.so.3). Both armhf legs pass end-to-end.

Notes / follow-ups

Deployment wiring (Terraform/Juju in the prod repo) is a separate change: reuse the arm64
OpenStack creds/project/network and set architecture = "armhf".
The arm64/armhf shared staging tenant (stg-ps7-...-arm64) has an external periodic resource
reaper that can transiently fail keypair assertions across arches; unrelated to this change.

Add "armhf"/"arm" as a buildable image architecture. armhf images are built on arm64 (aarch64) hosts running a 32-bit userspace and use the GitHub runner binary published by the canonical/github-actions-runner fork (actions-runner-linux-arm-<version>.tar.gz). Changes: - charm state.py: add ARM to the Arch enum and map the "armhf"/"arm"/"armv7l" config values to it in Arch.from_charm. - app config.py: add ARM to the Arch enum (drives the runner download arch "arm") with to_openstack() -> "armhf"; add ARM_ADDITIONAL_APT_PACKAGES (libicu74, libatomic1) required at runtime by the self-contained .NET runtime bundled in the linux-arm runner tarball. - app cloud_image.py: map Arch.ARM to the "armhf" Ubuntu cloud-image arch. - app openstack_builder.py: install the armhf runtime apt packages when building for Arch.ARM. - charmcraft.yaml: document the "armhf" architecture option. Unit tests added for every new arch mapping (TDD). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds armhf (32-bit ARM) as a supported build architecture across the charm and the image-builder app, enabling production of Ubuntu cloud images intended to run a 32-bit userspace on arm64 hardware.

Changes:

Extend charm-side architecture parsing to recognize armhf/arm/armv7l and map them to a new Arch.ARM.
Extend app-side architecture support to map Arch.ARM to OpenStack/cloud-image arch armhf, and install additional runtime apt packages for armhf builds.
Add/extend unit tests covering the new architecture mappings and cloud-init package selection.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`src/state.py`	Adds ARM arch recognition (`ARCHITECTURES_ARM`) and maps to `Arch.ARM`.
`tests/unit/test_state.py`	Extends charm unit tests for new ARM aliases.
`charmcraft.yaml`	Documents the new `armhf`/`arm` architecture option.
`app/src/github_runner_image_builder/config.py`	Adds `Arch.ARM`, OpenStack conversion to `armhf`, and ARM-specific apt package list.
`app/src/github_runner_image_builder/cloud_image.py`	Allows `armhf` as a supported Ubuntu cloud-image architecture and maps `Arch.ARM -> armhf`.
`app/src/github_runner_image_builder/openstack_builder.py`	Appends ARM additional apt packages in generated cloud-init for `Arch.ARM`.
`app/tests/unit/test_config.py`	Tests `Arch.ARM -> "armhf"` for OpenStack conversion.
`app/tests/unit/test_cloud_image.py`	Tests `Arch.ARM -> "armhf"` mapping for cloud-image selection.
`app/tests/unit/test_openstack_builder.py`	Tests ARM-specific additional apt packages in generated cloud-init.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Add rustup and docker-buildx to the armhf-only additional apt packages and set the default Rust toolchain for the ubuntu user during image build. - config.py: ARM_ADDITIONAL_APT_PACKAGES gains rustup and docker-buildx (docker.io is already in the default package set). - cloud-init.sh.j2: after configuring system users, run `sudo -u ubuntu rustup default stable`, guarded on the arm arch so it is a no-op on other architectures (rustup is only installed on armhf images). Test updated: the armhf cloud-init assertion now expects the extra packages and the rustup toolchain step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The armhf additional apt packages (libicu74, rustup, docker-buildx) are only available from the noble (24.04) archive onwards. On focal/jammy these package names do not exist, so an apt-get install would fail midway through the image build with a confusing error. Fail fast in _generate_cloud_init_script: when arch == ARM and the requested base image is older than noble, raise UnsupportedArchitectureError with a clear message instead of producing a cloud-init script that is guaranteed to fail. Tests: the cloud-init render test now exercises armhf on a supported (noble) base, and a new test asserts UnsupportedArchitectureError is raised for armhf on focal/jammy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…cript Add the missing Raises: section for the fail-fast guard that rejects armhf images on pre-noble base images, addressing review feedback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Breaks the commands.py <-> helpers.py circular import so command constants can be imported without the heavy OpenStack/fabric deps, enabling unit tests for arch-conditional command selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add ARM_RUNNER_COMMANDS (rustc/cargo, docker buildx, 32-bit Runner.Listener) and a pure commands_for_arch() selector, with unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Thread arch through run_openstack_tests so armhf images additionally assert rustup, docker-buildx, and the 32-bit runner binary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reuses the arm64 OpenStack tenant (creds/network/flavor) with --arch armhf. armhf is restricted to noble and resolute to match ARM_SUPPORTED_BASE_IMAGES. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The pytest --arch option restricted choices to amd64/arm64/s390x/ppc64le, so pytest rejected 'armhf' before arch_fixture could map it. Add armhf to the allowed choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Nova validates the glance image 'architecture' property against its architecture enum, which uses 'armv7l' for 32-bit ARM; 'armhf' (the Ubuntu userland ABI name) is rejected with 'Architecture name armhf is not valid'. The cloud-image download filename still uses 'armhf' via cloud_image.py, which is unaffected. Surfaced by the new armhf integration test when booting an OpenStack server from the built image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The armhf base image is tagged architecture=armv7l and booted on aarch64 compute hosts, which use the QEMU "virt" machine type. That machine type has no IDE bus, so libvirt defaulted to an IDE controller for the root disk and the config-drive CD-ROM and rejected the domain with "IDE controllers are unsupported for this QEMU binary or machine type", leaving the builder VM in ERROR. Set hw_machine_type=virt and virtio/scsi disk/cdrom buses on ARM image uploads so the guest boots. amd64 keeps the default PC machine type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

yhaliaw

This PR threads 32-bit ARM (armhf) support throughout the image builder stack — Arch.ARM, OpenStack image properties, ARM-specific apt packages, a rustup cloud-init step, and a base-image guard are all added consistently across the charm and app layers, with thorough unit and integration test coverage. The implementation is correct; the distinct arch strings (arm/armhf/armv7l) are intentional and round-tripped correctly across lookup points. The comments below are minor polish suggestions with no correctness implications.

🤝 Human review with AI assistance.

The armhf builder VMs booted on the aarch64 virt machine type but never became SSH-reachable. The root disk used virtio-blk (so the VM booted), but the config drive was attached as a virtio-scsi CD-ROM. The 32-bit armhf guest kernel lacks the virtio-scsi driver, so it could not read the config drive, cloud-init received no network metadata, and networking never came up. Attach the config drive on the same virtio-blk bus as the root disk and drop the now-unneeded virtio-scsi model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

virtio-blk cannot back ejectable CD-ROM media (libvirt rejects the domain with "disk type of 'vdb' does not support ejectable media"), so the config-drive CD-ROM must ride a virtio-scsi controller. This restores the bus layout that at least boots the guest so its serial console can be inspected to root-cause the SSH-unreachable behaviour on the aarch64 hosts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Native armhf VMs (32-bit kernel) do not boot on the ps7 arm64 hosts (empty console). The proven-working model from runner PRs #155/#169/#171 is an arm64 kernel running 32-bit armhf userland via native AArch32. This spec redefines the image-builder armhf path to build an arm64 boot image carrying the 32-bit linux-arm runner agent plus armhf multiarch userland. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The armhf runner image is an arm64 boot image carrying a 32-bit runner payload; native armhf (32-bit kernel) images do not boot on the aarch64 hosts. Schedule/boot the base image and snapshot as aarch64. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The builder VM must boot a 64-bit kernel on the aarch64 hosts. Download the arm64 base cloud image; the 32-bit runner payload is added via multiarch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

With the armhf runner image now built from an arm64 base (64-bit kernel), it boots with the cloud's normal defaults. The virtio-scsi/machine-type workaround for booting a native armhf (32-bit kernel) image is no longer needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The 32-bit linux-arm runner agent needs the armhf loader (libc6:armhf) and armhf builds of its .NET runtime deps (libicu74:armhf, libatomic1:armhf) to exec on the arm64 image via native AArch32. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add dpkg --add-architecture armhf (guarded to the arm runner) before apt install so the arm64 image can install and run the 32-bit linux-arm runner and its armhf runtime deps via native AArch32. armhf packages are served by ports.ubuntu.com, which the arm64 base already uses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The docs/superpowers plan and spec are internal agent planning artifacts, not product documentation. Untrack them and gitignore the directory so the Vale docs linter does not scan them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The arm64 cloud image's apt mirrors (clouds.archive/security.ubuntu.com) only carry the native architecture; armhf is a ports architecture and its package indices 404 there. Pin the stock sources to the native arch and add a ports.ubuntu.com source for armhf so the 32-bit runner's runtime deps (libc6:armhf, libicu74:armhf, libatomic1:armhf) resolve. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Two armhf package-selection bugs surfaced once the builder VM boots and runs apt: - rustup conflicts with the distro cargo/rustc packages ('held broken packages'). rustup provides the armhf/armv7 Rust toolchain, so drop cargo and rustc from the default apt set on armhf. - libicu's soname is release-specific (noble ships libicu74, resolute ships libicu78); the hardcoded libicu74:armhf is 'Unable to locate' on resolute. Select the armhf libicu matching the base image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The runner VM boots from a snapshot whose cloud-init state was cleared via "cloud-init clean --configs all" before snapshotting. On first boot cloud-init regenerates the stock apt sources, dropping the native-arch pin added by enable_armhf_multiarch, while the armhf dpkg architecture persists. apt-get update then requests binary-armhf indexes from the primary/security mirrors (which only serve arm64) and 404s, failing the runner's apt-get update. Bake an /etc/cloud/cloud.cfg.d drop-in with apt.preserve_sources_list: true so cloud-init keeps the pinned sources baked into the image. The drop-in lives outside /var/lib/cloud so it survives cloud-init clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

yanksyoon

This was a pretty complex task, thank you for making it happen @florentianayuwono !

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot added the Libraries: OK label Jul 2, 2026

florentianayuwono requested a review from Copilot July 2, 2026 07:19

Copilot started reviewing on behalf of florentianayuwono July 2, 2026 07:20 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread tests/unit/test_state.py

Comment thread charmcraft.yaml Outdated

Comment thread app/src/github_runner_image_builder/openstack_builder.py Outdated

florentianayuwono and others added 3 commits July 2, 2026 16:21

Apply suggestions from code review

30daea5

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

florentianayuwono marked this pull request as ready for review July 2, 2026 10:51

florentianayuwono requested review from cbartz, javierdelapuente, yanksyoon and yhaliaw as code owners July 2, 2026 10:51

cbartz reviewed Jul 2, 2026

View reviewed changes

Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2

Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2

yhaliaw reviewed Jul 3, 2026

View reviewed changes

Comment thread app/src/github_runner_image_builder/openstack_builder.py

florentianayuwono and others added 9 commits July 6, 2026 16:50

docs: document UnsupportedArchitectureError in _generate_cloud_init_s…

63a87cc

…cript Add the missing Raises: section for the fail-fast guard that rejects armhf images on pre-noble base images, addressing review feedback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test: add arch-conditional armhf runner assertions

05bca9e

Add ARM_RUNNER_COMMANDS (rustc/cargo, docker buildx, 32-bit Runner.Listener) and a pure commands_for_arch() selector, with unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test: run arch-specific commands in openstack integration test

9887b4d

Thread arch through run_openstack_tests so armhf images additionally assert rustup, docker-buildx, and the 32-bit runner binary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test: map --arch armhf to Arch.ARM in integration conftest

6f1dce3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: add armhf integration-test leg on noble+resolute

e73ce4c

Reuses the arm64 OpenStack tenant (creds/network/flavor) with --arch armhf. armhf is restricted to noble and resolute to match ARM_SUPPORTED_BASE_IMAGES. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

yhaliaw approved these changes Jul 8, 2026

View reviewed changes

Comment thread app/src/github_runner_image_builder/config.py Outdated

florentianayuwono and others added 2 commits July 8, 2026 18:48

Merge branch 'main' into feat/armhf-arch-support-isd-5856

1ddc559

florentianayuwono self-assigned this Jul 9, 2026

florentianayuwono marked this pull request as draft July 9, 2026 12:26

florentianayuwono and others added 11 commits July 9, 2026 21:05

fix(armhf): download arm64 base cloud image for Arch.ARM

7d19b97

The builder VM must boot a 64-bit kernel on the aarch64 hosts. Download the arm64 base cloud image; the 32-bit runner payload is added via multiarch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs(armhf): add arm64 multiarch image implementation plan

90e0f06

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

florentianayuwono marked this pull request as ready for review July 10, 2026 08:10

yanksyoon approved these changes Jul 10, 2026

View reviewed changes

chore: bump copyright year to 2026 in test_integration_commands

3eb7dd4

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

florentianayuwono enabled auto-merge (squash) July 10, 2026 11:27

florentianayuwono merged commit b60a2bd into main Jul 11, 2026
74 of 91 checks passed

florentianayuwono deleted the feat/armhf-arch-support-isd-5856 branch July 11, 2026 05:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add armhf (32-bit ARM) architecture support [ISD-5856]#233

feat: add armhf (32-bit ARM) architecture support [ISD-5856]#233
florentianayuwono merged 28 commits into
mainfrom
feat/armhf-arch-support-isd-5856

florentianayuwono commented Jul 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yhaliaw left a comment

Uh oh!

Uh oh!

yanksyoon left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

florentianayuwono commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

How an armhf image is built

Why preserve_sources_list: true

armhf is restricted to noble+ base images

Changes

Tests

Notes / follow-ups

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yhaliaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yanksyoon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

florentianayuwono commented Jul 2, 2026 •

edited

Loading

Why `preserve_sources_list: true`